A neural network is one example of a parallel, distributed, data processing system. A neural network contains a large number of processing elements or nodes of simple processing capability, the number of nodes being the size of the network. The nodes are connected with each other to form a fully connected, or nearly fully connected network. The network performs parallel data processing based on a constraint satisfaction paradigm that has been shown to lead to collective computation capabilities.
The operation of neural networks can be modeled as an iterative process. Each node has a particular value associated with it. In each iteration, each node updates its value based upon the values of the other nodes, the update operation typically consisting of a matrix-vector multiplication. The update algorithm reflects the influences on each node of the other nodes in the network. The matrix-vector multiplication is often followed by some nonlinear function, such as clipping or clamping.
One of the many applications of a neural network is as an associative memory system. In this particular application, once a set of patterns, termed "library elements", has been stored in the network, the user can recall a desired pattern or library element by providing the network with incomplete or noisy information on the pattern in question. For example, if the library elements comprise a set of two-dimensional images, one can input a portion of one image to the neural network, and the neural network will produce the entire image. An associative memory based on a neural network is robust and fault tolerant. However, the neural network requires a separate processor for each node. In the large systems required for practical applications, this can lead to considerable difficulties in implementing the network.